Living Hand to Mouth: Psychological Theories about Speech and Gesture in Interactive Dialogue Systems

In this paper we discuss the application of aspects of a psychological theory about the relationship between speech and gesture to the implementation of interactive dialogue systems. We first lay out some uncontroversial facts about the interaction of speech and gesture in conversation and describe some psychological theories put forth to explain those data, settling on one theory as being the most interesting for interactive dialogue systems. We then lay out our implementation of an interactive dialogue system that is informed by the theory— concentrating on two particular claims of the theory: that gesture and speech reflect a common conceptual source; and that the content and form of gesture is tuned to the communicative context and the actor’s communicative intentions. We compare our work to some other similar interactive systems, and conclude with some thoughts about how both implementation and theory can benefit from this kind of close partnership.